Learning assistance device, method of operating learning assistance device, learning assistance program, learning assistance system, and terminal device

ABSTRACT

A learning assistance device acquires a plurality of learned discriminators obtained by causing learning discriminators provided in a plurality of respective terminal devices to perform learning using image correct answer data, acquires a plurality of discrimination results obtained by causing a plurality of learned discriminators to discriminate the same input image, determines the correct answer data of the input image on the basis of the plurality of discrimination results, causes the discriminator to perform learning the input image and the correct answer data, and outputs a result thereof as a new learning discriminator to each terminal device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Divisional of copending U.S. patent applicationSer. No. 16/123,456 filed Sep. 6, 2018, which claims priority under 35U.S.C. § 119(a) to Japanese Patent Application No. 2017-187096, filed onSep. 27, 2017, all of which are hereby expressly incorporated in theirentirety by reference into the present application.

BACKGROUND Field of the Invention

The present invention relates to a learning assistance device thatassists in generation of a discriminator using machine learning, amethod of operating the learning assistance device, a learningassistance program, a learning assistance system, and a terminal device.

Related Art

In the related art, machine learning has been used to learn features ofdata and perform recognition or classification of images or the like. Inrecent years, various learning schemes have been developed, a processingtime has been shortened due to an improved processing capability of acomputer, and deep learning in which a system learns features of imagedata or the like at a deeper level can be performed. By performing thedeep learning, features of images or the like can be recognized withvery high accuracy, and improvement of performance of discrimination isexpected.

In the medical field, artificial intelligence (AI) that recognizesfeatures of images with high accuracy by performing learning using deeplearning is desired. For deep learning, it is indispensable to performlearning using a large amount of high quality data according topurposes. Therefore, it is important to prepare learning dataefficiently. Image data of a large number of cases of disease isaccumulated in each medical institution with the spread of a picturearchiving and communication system (PACS). Therefore, learning usingimage data of various cases of disease accumulated in each medicalinstitution is considered.

Further, in recent years, a technical level of artificial intelligencehas been improving in a variety of fields, and the artificialintelligence is incorporated into a variety of services and started tobe used and utilized. In particular, services provided to various edgeterminals over a network are increasing. For example, JP2008-046729Adiscloses a device in which a learning model is incorporated into amoving image topic division device that automatically divides a movingimage at a switching point of a topic. JP2008-046729A discloses that thelearning model has been distributed to a client terminal, the topicdivision is automatically executed using the distributed learning modelat each client terminal, and content corrected by a user for a result ofthe automatic topic division is fed back for updating of the learningmodel. After the feedback corrected by the user is accumulated in anintegration module over a network, a learning model reconstructed usingthe accumulated feedback is distributed to each client terminal over thenetwork again.

However, in the medical field, since data to be learned is medical dataof a patient, confidentiality is very high, and it is necessary tohandle data carefully for use as learning data. Further, correct answerdata is not attached to image data. Alternatively, even in a case wherethere is the correct answer data, the correct answer data is oftenmanaged without being associated with original image data. Therefore, itis difficult and costly to efficiently collect learning data to whichthe correct answer data is added.

SUMMARY

Therefore, in order to solve the above-described problems, an object ofthe present invention is to provide a learning assistance device whichenables learning of a large amount and a variety of learning datanecessary for deep learning in a medical field, a method of operatingthe learning assistance device, a learning assistance program, alearning assistance system, and a terminal device.

A learning assistance device of the present invention comprises: alearned discriminator acquisition unit that acquires a plurality oflearned discriminators obtained by causing each of learningdiscriminators provided in a plurality of respective terminal devices toperform learning using an image and correct answer data thereof; and adiscriminator output unit that acquires a plurality of discriminationresults obtained by causing each of the plurality of learneddiscriminators to discriminate the same input image, determines correctanswer data of the input image on the basis of the plurality ofdiscrimination results, and outputs a new learning discriminatorobtained by causing the learning discriminator to perform learning againusing the input image and the determined correct answer data, whereinthe learning assistance device repeatedly performs a process in whichthe learning discriminator acquisition unit acquires the plurality oflearned discriminators obtained by causing the each of learningdiscriminators, the each of learning discriminators being output fromthe discriminator output unit and being provided for each of theplurality of terminal devices, to perform learning using an image andcorrect answer data thereof, and the discriminator output unitdetermines a new correct answer data of a new same input image differentfrom the input image on the basis of a plurality of discriminationresults obtained by causing the plurality of learned discriminatorsacquired by the learned discriminator acquisition unit to discriminatethe new same input image, and outputs a new learning discriminatorobtained by causing the learning discriminator output by thediscriminator output unit to perform learning again using the new inputimage and the determined new correct answer data.

A method of operating a learning assistance device of the presentinvention is a method of operating a learning assistance deviceincluding a learned discriminator acquisition unit and a discriminatoroutput unit, the method comprising: acquiring a plurality of learneddiscriminators obtained by causing each of learning discriminatorsprovided in a plurality of respective terminal devices to performlearning using an image and correct answer data thereof by the learneddiscriminator acquisition unit; acquiring a plurality of discriminationresults obtained by causing each of the plurality of learneddiscriminators to discriminate the same input image, determines correctanswer data of the input image on the basis of the plurality ofdiscrimination results, and outputs a new learning discriminatorobtained by causing the learning discriminator to perform learning againusing the input image and the determined correct answer data by thediscriminator output unit; and repeatedly performing a process in whichthe learned discriminator acquisition unit acquires the plurality oflearned discriminators obtained by causing the each of learningdiscriminators, the each of learning discriminators being output fromthe discriminator output unit and being provided for each of theplurality of terminal devices, to perform learning using an image andcorrect answer data thereof, and the discriminator output unitdetermines a new correct answer data of a new same input image differentfrom the input image on the basis of a plurality of discriminationresults obtained by causing the plurality of learned discriminatorsacquired by the learned discriminator acquisition unit to discriminatethe new same input image, and outputs a new learning discriminatorobtained by causing the learning discriminator output by thediscriminator output unit to perform learning again using the new inputimage and the determined new correct answer data.

A learning assistance program according to the present invention causesa computer to function as: a learned discriminator acquisition unit thatacquires a plurality of learned discriminators obtained by causing eachof learning discriminators provided in a plurality of respectiveterminal devices to perform learning using an image and correct answerdata thereof; and a discriminator output unit that acquires a pluralityof discrimination results obtained by causing each of the plurality oflearned discriminators to discriminate the same input image, determinescorrect answer data of the input image on the basis of the plurality ofdiscrimination results, and outputs a new learning discriminatorobtained by causing the learning discriminator to perform learning againusing the input image and the determined correct answer data, whereinthe program causes a process in which the learned discriminatoracquisition unit acquires the plurality of learned discriminatorsobtained by causing the each of learning discriminators, the each oflearning discriminators being output from the discriminator output unitand being provided for each of the plurality of terminal devices, toperform learning using an image and correct answer data thereof, and thediscriminator output unit determines a new correct answer data of a newsame input image different from the input image on the basis of aplurality of discrimination results obtained by causing the plurality oflearned discriminators acquired by the learned discriminator acquisitionunit to discriminate the new same input image, and outputs a newlearning discriminator obtained by causing the learning discriminatoroutput by the discriminator output unit to perform learning again usingthe new input image and the determined new correct answer data, to berepeatedly performed.

“Acquire a learned discriminator” may be acquiring the learneddiscriminator or may be receiving a parameter of the discriminator andsetting the parameter in a prepared discriminator to acquire the learneddiscriminator. For example, in a case where the discriminator is amultilayered neural network, a program in which a learned multilayeredneural network is incorporated may be acquired or a weight of couplingbetween layers of units of the multilayered neural network may beacquired as a parameter and the parameter may be set in the preparedmultilayered neural network, so that the learned multilayered neuralnetwork can be acquired.

Further, the discriminator output unit further outputs an actuallyoperated discriminator learning the image and the correct answer datathereof used for learning by the new learning discriminator.

The “actually operated discriminator” is a discriminator capable ofacquiring a discrimination result of input image data, which cannotperform additional learning, and the “learning discriminator” is adiscriminator that can perform additional learning using an image andcorrect answer data of the image. Further, the actually operateddiscriminator to be output is a discriminator caused to perform learningusing an image and correct answer data of the image that are all thesame as the image and the correct answer data of the image learned bythe output learning discriminator.

Further, the learned discriminator acquisition unit may acquire thelearned discriminator from the plurality of terminal devices over anetwork, and the discriminator output unit may output the new learningdiscriminator to the plurality of terminal devices over the network.

Further, the discriminator output unit may determine a discriminationresult having the largest number of same results among the plurality ofdiscrimination results, as correct answer data of the input image.

Further, the discriminator output unit may determine a weight of each ofthe plurality of learned discriminators according to the terminal devicelearned by the learned discriminator, adds the weights of the learneddiscriminators having the same result among the discrimination results,and sets a discrimination result having the largest added weight ascorrect answer data of the input image.

Further, the discriminator output unit may determine the weight of eachof the learned discriminators learned at each of the terminal devicesaccording to the number of pieces of correct answer data learned by thelearned discriminator at each terminal device, add the weights of thelearned discriminators having the same discrimination result, and set adiscrimination result having the largest added weight as correct answerdata of the input image.

Further, the discriminator output unit may determine weights for typesof cases of disease of the image learned by the respective learneddiscriminators with respect to the respective learned discriminators,add the weights corresponding to the types of cases of disease of theinput image in the learned discriminator having the same discriminationresult, and set a discrimination result having the largest added weightas correct answer data of the input image.

Further, the learning assistance device may further comprise anevaluation unit that evaluates a correct answer rate using an image setincluding a plurality of images serving as a reference with respect tothe plurality of learned discriminators, wherein the discriminatoroutput unit may determine the weight of each of the plurality of learneddiscriminators according to the correct answer rate obtained by theevaluation unit, add the weights of the respective learneddiscriminators having the same discrimination results, and set thediscrimination result having the largest added weight as correct answerdata of the input image.

A learning assistance system according to the present invention is alearning assistance system in which a learning assistance device and aplurality of terminal devices are connected over a network, wherein theterminal device includes a learned discriminator output unit thatoutputs a learned discriminator obtained by causing a learningdiscriminator to perform learning using an image and correct answer datathereof over the network, the learning assistance device includes alearned discriminator acquisition unit that acquires a plurality oflearned discriminators from the plurality of terminal devices over thenetwork, and a discriminator output unit that acquires a plurality ofdiscrimination results obtained by causing the plurality of learneddiscriminators to discriminate the same input image, determines correctanswer data of the input image on the basis of the plurality ofdiscrimination results, and outputs a new learning discriminatorobtained by causing the learning discriminator to perform learning againusing the input image and the determined correct answer data to theplurality of terminal devices over the network, and the terminal deviceincludes a discriminator acquisition unit that receives the learningdiscriminator output from the learning assistance device over thenetwork.

Further, in the learning assistance system, the discriminatoracquisition unit may further acquire an actually operated discriminatorlearning the same image and correct answer data of the image as those ofthe learning discriminator output from the learning assistance deviceover a network, and the terminal device may further include adiscrimination result acquisition unit that acquires a discriminationresult of discriminating an image that is a discrimination target usingthe actually operated discriminator.

A terminal device according to the present invention comprises adiscriminator acquisition unit that acquires a learning discriminator,and a learned actually operated discriminator learned using the sameimage and correct answer data of the image as those of the learningdiscriminator; a discrimination result acquisition unit that acquires adiscrimination result of discriminating an image that is adiscrimination target using the actually operated discriminator; and alearned discriminator output unit that outputs a learned discriminatorobtained by causing the learning discriminator to perform learning usingan image and correct answer data thereof.

Further, in the terminal device, the discriminator acquisition unit mayacquire the learning discriminator and the actually operateddiscriminator from a learning assistance device over a network, and thelearned discriminator output unit may send and output the learneddiscriminator over the network.

Another learning assistance device of the present invention comprises amemory that stores instructions to be executed by a computer, and aprocessor configured to execute the stored instructions, wherein theprocessor executes an acquisition process of acquiring a plurality oflearned discriminators obtained by causing each of learningdiscriminators provided in a plurality of respective terminal devices toperform learning using an image and correct answer data thereof, and anoutput process of acquiring a plurality of discrimination resultsobtained by causing each of the plurality of learned discriminators todiscriminate the same input image, determining correct answer data ofthe input image on the basis of the plurality of discrimination results,and outputting a new learning discriminator obtained by causing thelearning discriminator to perform learning again using the input imageand the determined correct answer data, and the learning assistancedevice repeatedly performs a process of acquiring the plurality oflearned discriminators obtained by causing the each of learningdiscriminators, the each of learning discriminators being output fromthe discriminator output unit and being provided for each of theplurality of terminal devices, to perform learning using an image andcorrect answer data thereof, determining a new correct answer data of anew same input image different from the input image on the basis of aplurality of discrimination results obtained by causing the plurality oflearned discriminators acquired by the learned discriminator acquisitionunit to discriminate the new same input image, and outputting a newlearning discriminator obtained by causing the learning discriminatoroutput by the discriminator output unit to perform learning again usingthe new input image and the determined new correct answer data.

According to the present invention, the learning assistance deviceacquires the plurality of learned discriminators obtained by causinglearning discriminators provided in a plurality of respective terminaldevices to perform learning using image correct answer data, acquiresthe plurality of discrimination results of discriminating the same inputimage, determines the correct answer data of the input image on thebasis of the plurality of discrimination results, and causes thediscriminator to perform learning using the input image and thedetermined correct answer data. Therefore, it is possible toautomatically generate the data for learning from the image to which thecorrect answer data is not attached, perform deep learning using a largeamount of images, and improve performance of discrimination.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a schematic configuration of a learningassistance system of the present invention.

FIG. 2 is a diagram illustrating a schematic configuration of a medicalinformation system.

FIG. 3 illustrates an example of a multilayered neural network.

FIG. 4 is a block diagram illustrating a schematic configuration of aterminal device and a learning assistance device according to a firstembodiment.

FIG. 5 is a diagram illustrating learning of a discriminator.

FIG. 6 is a flowchart showing a flow of a process of causing thediscriminator to perform learning.

FIG. 7 is a block diagram illustrating a schematic configuration of aterminal device and a learning assistance device according to a secondembodiment.

FIG. 8 is a block diagram illustrating a schematic configuration of aterminal device and a learning assistance device according to a thirdembodiment.

FIG. 9 is a block diagram illustrating a schematic configuration of aterminal device and a learning assistance device according to a fourthembodiment.

DETAILED DESCRIPTION

FIG. 1 illustrates a schematic configuration of a learning assistancesystem 1 according to a first embodiment of the present invention. Thelearning assistance system 1 is configured by connecting a plurality ofterminal devices 10 installed in a plurality of medical institutions A,B, . . . , X and a learning assistance device 20 placed on a cloud sideover a network 30.

The learning assistance device 20 includes a well-known hardwareconfiguration such as a central processing unit (CPU), a memory, astorage, an input and output interface, a communication interface, aninput device, a display device, and a data bus, and is ahigh-performance computer in which a well-known operation system or thelike is installed and which has a server function. Further, a graphicsprocessing unit (GPU) may be provided, as necessary. Alternatively, thelearning assistance device 20 may be a virtualized virtual serverprovided using one or a plurality of computers. The learning assistanceprogram of the present invention is installed in a server, and functionsas a learning assistance device by a program instruction being executedby the CPU of the computer.

The terminal device 10 is a computer for image processing provided inthe respective medical institutions A, B, . . . , X, and includes awell-known hardware configuration such as a CPU, a memory, a storage, aninput and output interface, a communication interface, an input device,a display device, and a data bus. A well-known operation system or thelike is installed in the terminal device 10. The terminal device 10includes a display as a display device. Further, a GPU may be provided,as necessary.

The network 30 is a wide area network (WAN) that widely connects theterminal devices 10 placed at the plurality of medical institutions A,B, . . . , X to the learning assistance device 20 via a public networkor a private network.

Further, as illustrated in FIG. 2, the terminal device 10 is connectedto respective medical information systems 50 of the respective medicalinstitutions A, B, . . . , X over a local area network (LAN) 51. Themedical information system 50 includes a modality (an imaging device)52, an image database 53, and an image interpretation medicalworkstation 54, and is configured so that transmission and reception ofimage data to and from each other are performed over the network 51. Itshould be noted that in the network 51, it is desirable to use acommunication cable such as an optical fiber so that image data can betransferred at a high speed.

The modality 52 includes a device that images an examination target partof a subject to generate an examination image representing the part,adds accessory information defined in a DICOM standard to the image, andoutputs the resultant image. Specific examples of the device include acomputed tomography (CT) device, a magnetic resonance imaging (MRI)device, a positron emission tomography (PET) device, an ultrasonicdevice, and a computed radiography (CR) device using a planar X-raydetector (FPD: flat panel detector).

In the image database 53, a software program for providing a function ofa database management system (DBMS) is incorporated in a general-purposecomputer, and a large capacity storage is included. This storage may bea large capacity hard disk device, or may be a disk device connected toa network attached storage (NAS) or a storage area network (SAN)connected to the network 51. Further, the image data captured by themodality 52 is transmitted to and stored in the image database 53 overthe network 51 according to a storage format and a communicationstandard conforming to a DICOM standard.

The image interpretation medical workstation 54 is a computer that isused for an image interpretation doctor of a radiology department tointerpret an image and create an interpretation report. The imageinterpretation medical workstation 54 performs a display of the imagedata received from the image database 53 and performs automaticdetection of a portion likely to be a lesion in the image.

In the embodiment, a case where an image processing program in which adiscriminator functioning as an actually operated discriminator isincorporated in each terminal device 10 is provided from the learningassistance device 20, and a learning program in which a discriminatorfunctioning as a learning program separately from the image processingprogram is incorporated is provided will be described. The imageprocessing program and the learning program distributed to each terminaldevice 10 are installed in the terminal device 10 to function as animage processing device in which the actually operated discriminator isincorporated, and a learning discriminator.

Further, a case where the actually operated discriminator and thelearning discriminator are multilayered neural networks subjected todeep learning to be able to discriminate a plurality of types of organareas and/or lesion areas will be described. In the multilayered neuralnetwork, a calculation process is performed on a plurality of pieces ofdifferent calculation result data obtained by a preceding layer forinput data, that is, extraction result data of a feature amount usingvarious kernels in each layer, data of the feature amount obtained bythe calculation process is acquired, and a further calculation processis performed on the data of the feature amount in the next andsubsequent processing layers. Thus, it is possible to improve arecognition rate of the feature amount and to discriminate which of aplurality of types of areas the input image data is.

FIG. 3 is a diagram illustrating an example of the multilayered neuralnetwork. As illustrated in FIG. 3, the multilayered neural network 40includes a plurality of layers including an input layer 41 and an outputlayer 42. In FIG. 3, a layer before the output layer 42 is denoted byreference numeral 43.

In the multilayered neural network 40, the image data is input to theinput layer 41 and a discrimination result of an area is output. In acase where learning is performed, the output discrimination result iscompared with correct answer data, and a weight of coupling between therespective layers of units (indicated by circles in FIG. 3) included inthe respective layers of the multilayered neural network 40 is correctedfrom the output side (the output layer 42) to the input side (the inputlayer 41) according to whether an answer is a correct answer or anincorrect answer. The correction of the weight of coupling is repeatedlyperformed a predetermined number of times, or is repeatedly performeduntil a correct answer rate of the output discrimination result is 100%or is equal to or greater than a predetermined threshold value using alarge number of pieces of image data with correct answer data, and thelearning ends.

FIG. 4 is a block diagram illustrating a schematic configuration of theterminal device 10 and the learning assistance device 20. Functions ofthe terminal device 10 and the learning assistance device 20 will bedescribed in detail with reference to FIG. 4. First, the terminal device10 will be described.

The terminal device 10 includes a discriminator acquisition unit 12, adiscrimination result acquisition unit 13, a learning unit 14, and alearned discriminator output unit 15.

The discriminator acquisition unit 12 acquires a learning discriminatorand an actually operated discriminator. For example, the imageprocessing program and the learning program are received from thelearning assistance device 20 over the network 30, and the receivedimage processing program is installed. Accordingly, image processing inwhich the actually operated discriminator is incorporated becomesexecutable in the terminal device 10 and functions as the discriminationresult acquisition unit 13. Similarly, the learning program isinstalled, and the learning discriminator becomes executable andfunctions as the learning unit 14. It should be noted that the learningdiscriminator is a discriminator that has learned the same image correctanswer data as the actually operated discriminator received from thelearning assistance device 20. In the following description, the imageprocessing in which the actually operated discriminator is incorporatedis simply referred to as an actually operated discriminator. It shouldbe noted that the image correct answer data refers to a combination ofthe image data and correct answer data thereof. Details of the imagecorrect answer data will be described below.

The discrimination result acquisition unit 13 inputs a discriminationtarget image data to the actually operated discriminator and acquires adiscrimination result. The actually operated discriminator is adiscriminator of which discrimination performance has been guaranteed inthe learning assistance device 20, and in each of the medicalinstitutions A, B, . . . , X, the discrimination is performed on theimage data that is a diagnosis target using the actually operateddiscriminator. Further, the discrimination result acquisition unit 13may perform discrimination of the image data that is a diagnosis targetsent from the image interpretation medical workstation 54 to theterminal device 10 over the network 51, and transmit a discriminationresult from the terminal device 10 to the image interpretation medicalworkstation 54.

The learning unit 14 causes the learning discriminator to performlearning using the image data and the correct answer data thereof. Thecorrect answer data includes a mask image showing an area such as anorgan or abnormal shadow of the image data, and information indicatingwhat the area of the mask image is (for example, an area of an organsuch as a liver, a kidney, or a lung or an area of an abnormal shadowsuch as a liver cancer, a kidney cancer, or a pulmonary nodule).

The correct answer data may be created by an image interpretation doctoror the like of each of the medical institutions A, B, . . . , Xobserving the image data. For example, the image data is extracted fromthe image database 53, the discrimination result acquisition unit 13inputs the image data to the actually operated discriminator andacquires a discrimination result, and a user such as the imageinterpretation doctor determines whether the discrimination result is acorrect answer or an incorrect answer, and stores a discriminationresult together with the input image data and correct answer data in theimage database 53 as image correct answer data in the case of thecorrect answer. In the case of an incorrect answer, the user generates amask image of the correct answer data, assigns the correct answer datato the image data, and stores the resultant data in the image database53 as image correct answer data.

Therefore, the learning unit 14 causes the multilayered neural network40 of the learning discriminator to perform learning using a largenumber of pieces of image correct answer data stored in the imagedatabase 53. First, the image data of the image correct answer data isinput to the multilayered neural network 40, and a discrimination resultis output. Then, the output discrimination result is compared with thecorrect answer data, and a weight of coupling between the respectivelayers of the units included in the respective layers of themultilayered neural network 40 from the output side to the input side iscorrected according to whether the answer is a correct answer or anincorrect answer. The correction of the weight of the coupling isrepeatedly performed using a large number of pieces of correct answerdata a predetermined number of times or until the correct answer rate ofthe output discrimination result becomes 100%, and the learning isended.

The learned discriminator output unit 15 outputs the learningdiscriminator of which the learning has ended in the learning unit 14 asa learned discriminator. Specifically, the weight (hereinafter referredto as a parameter) of coupling between the layers of the unitsconstituting the neural network constituting the learned discriminatoris periodically transmitted to the learning assistance device 20 overthe network 30.

Next, the learning assistance device 20 will be described. Asillustrated in FIG. 4, the learning assistance device 20 includes alearned discriminator acquisition unit 22, a discriminator storage unit23, a correct answer data acquisition unit 24, a correct answer datastorage unit 25, a learning unit 26, and a discriminator output unit 27.

The learned discriminator acquisition unit 22 receives a parameter ofthe multilayered neural network 40 constituting the learneddiscriminators transmitted from the plurality of terminal devices 10over the network 30. The received parameter is temporarily stored in thediscriminator storage unit 23. The multilayered neural network 40 isprovided in the learning assistance device 20 in advance and theparameter received from each terminal device 10 is set as the weight ofthe coupling between the respective layers of the units of themultilayered neural network 40 provided in the learning assistancedevice 20. By re-setting the parameter received from each terminaldevice in this weight, the same learned discriminator as each terminaldevice 10 can be acquired.

The correct answer data acquisition unit 24 causes each of the learneddiscriminators collected from each of the terminal devices 10 todiscriminate the same input image data to acquire a plurality ofdiscrimination results, and determines the correct answer data of theinput image data from the plurality of discrimination results.

A large number of pieces of image data are often stored in a databasewithout correct answer data attached thereto. In order to attach thecorrect answer data to the image data, for example, the image data isinput to the discriminator so as to acquire a discrimination result, anda user such as an image interpretation doctor performs a determinationthat the answer is a correct answer or an incorrect answer with respectto the discrimination result, and registers the discrimination result asthe image correct answer data in association with the correct answerdata and the input image data in the case of the correct answer. In caseof the incorrect answer, the user creates a mask image of the correctanswer data and registers the mask image as image correct answer data inassociation with input image data. Work of creating the correct answerdata in this way is laborious and it is difficult to manually generate alarge number of pieces of correct answer data.

Therefore, the correct answer data acquisition unit 24 determines thelargest number of same discrimination results as correct answer data ofthe input image data from the discrimination results obtained byinputting the same input image data to the learned discriminatorscollected from the respective terminal devices 10. Thus, in a case wherethe correct answer data is determined from a plurality of discriminationresults obtained by using the learned discriminators collected from therespective terminal devices 10, accurate correct answer data can beautomatically generated, and therefore, it is possible to easily obtaina large number of pieces of correct answer data. The obtained correctanswer data is accumulated in the storage (the correct answer datastorage unit 25) as image correct answer data in association with theinput image data.

The learning unit 26 is provided with a deep learning discriminatorconfigured of a multilayered neural network 40. The image correct answerdata accumulated in the correct answer data storage unit 25 issequentially input to the deep learning discriminator, so that learningis performed.

In a stage the learning had progressed to a certain extent and accuracyof the discrimination of the deep learning discriminator has improved,the discriminator output unit 27 generates an image processing program(actually operated discriminator) incorporating the deep learningdiscriminator learned by the learning unit 26 and a learning program(the learning discriminator), and distributes the programs to eachterminal device 10 over the network 30. Since software intended formedical purposes is a target of the Pharmaceutical and Medical DeviceAct (the revised Pharmaceutical Affairs Act), the software is requiredto meet a criterion prescribed in the Pharmaceutical and Medical DeviceAct. Therefore, it is preferable to confirm that a deep learningdiscriminator before distribution exceeds an evaluation standard usingan evaluation image set formed through a combination of a plurality ofimages in which the criterion prescribed in the Pharmaceutical andMedical Device Act can be evaluated, and then, distribute deep learningdiscriminator.

Even after the discriminator output unit 27 distributes the imageprocessing program and the learning program to the respective terminaldevices 10, the learning unit 26 sequentially inputs the image correctanswer data accumulated in the correct answer data storage unit 25 tothe deep learning discriminator as it is and causes the deep learningdiscriminator to perform learning. That is, the learning unit 26 causesthe learning discriminator output by the discriminator output unit 27 toperform additional learning, and the discriminator output unit 27outputs the additionally learned learning discriminator as a newlearning discriminator.

Next, a flow of a deep learning process of the embodiment will bedescribed with reference to a transition diagram of FIG. 5 and aflowchart of FIG. 6.

First, in the learning assistance device 20, the discriminator outputunit 27 distributes the actually operated discriminator NNo and thelearning discriminator NNt to the plurality of terminal devices 10, andto the medical institution A, . . . , the medical institution X over thenetwork 30 (S1).

In the terminal device 10, the discriminator acquisition unit 12acquires the actually operated discriminator NNo and the learningdiscriminator NNt (S2). The actually operated discriminator NNo is usedfor diagnosis by an image interpretation doctor, and the discriminationresult acquisition unit 13 discriminates image data (input) that is adiagnosis target and obtains a discrimination result (output) (see FIG.5). Further, in the medical institution A, the learning unit 14 of theterminal device 10 causes the learning discriminator NNt to performlearning using the image correct answer data T stored in the imagedatabase 53 (S3) and generates the learned discrimination unit NNt-A(S4). Similarly, in the medical institution X, the learning unit 14 ofthe terminal device 10 causes the learning discriminator NNt to performlearning using the image correct answer data T and generates the learneddiscriminator NNt-X (S4).

Periodically, in each terminal device 10, the learned discriminatoroutput unit 15 transmits the learned discriminator to the learningassistance device 20 (S5). The parameter of the learned discriminatorNNt-A is transmitted from the terminal device 10 of the medicalinstitution A to the learning assistance device 20, and the parameter ofthe learned discriminator NNt-X is transmitted from the terminal device10 of the medical institution X to the learning assistance device 20(see a solid arrow (1) in FIG. 5).

In the learning assistance device 20, the learned discriminatoracquisition unit 22 temporarily stores the parameters of the learneddiscriminators received from the plurality of terminal devices 10 in thediscriminator storage unit 23. By setting this parameter in themultilayered neural network 40 provided in the learning assistancedevice 20, the learned discriminator learned by each terminal device 10is acquired (S6).

The correct answer data acquisition unit 24 inputs the input image dataP to the learned discriminator of each terminal device 10 to obtain thediscrimination result. In the example of FIG. 5, a result a, a result b,a result c, . . . , a result g are obtained, and a largest number ofresults b are determined to be correct answer data of the input imagedata P (S7). The input image data P and the correct answer data areaccumulated in the storage 25 in association with each other.

The learning unit 26 causes the deep learning discriminator NNl toperform learning using the input image data P and the result b (correctanswer data) (S8). Periodically, a new version of the actually operateddiscriminator NNo and the learning discriminator NNt are generated onthe basis of the deep learning discriminator NNl (S9). The discriminatoroutput unit 27 distributes the new version of the actually operateddiscriminator NNo and learning discriminator NNt to each of the terminaldevices 10 (S10; see an arrow (2) of a broken line in FIG. 5).

In the terminal device 10 of the medical institution A, thediscriminator acquisition unit 12 acquires the new version of theactually operated discriminator NNo and learning discriminator NNt again(S2). The learning unit 14 of the terminal device 10 causes the newversion of learning discriminator NNt to perform learning using theimage correct answer data T stored in the image database 53 (S3) andgenerate the learned discriminator NNt-A again (S4).

In the terminal device 10 of the medical institution X, thediscriminator acquisition unit 12 acquires the new version of theactually operated discriminator NNo and learning discriminator NNt again(S2). The learning unit 14 of the terminal device 10 causes the newversion of learning discriminator NNt to perform learning using theimage correct answer data T stored in the image database 53 (S3) andgenerates a learned discriminator NNt-X (S4). Consequently, the processfrom S5 to S10 is performed, as in the same manner as described above.

The processes of S2 to S10 are repeated, and the learning assistancedevice 20 generates an actually operated discriminator and a learningdiscriminator of which the performance has been improved whilegenerating the correct answer data, and distributes the actuallyoperated discriminator and the learning discriminator to the terminaldevice 10.

As described above, by the terminal device 10 placed in each medicalinstitution performing learning using the image data stored in themedical institution, a discriminator improving discriminationperformance is generated in each medical institution. By the learningassistance device 20 generating the correct answer data using thediscriminators of which the performance have been improved in eachmedical institution, a large amount of accurate correct answer data canbe generated, and deep Learning can be performed using this correctanswer data.

Although the case where the mask image of the correct answer data forthe image data has the information indicating what the area on the imageis has been described above, the discriminator may be configured to (1)determine an organ area and an organ name by determining what organ eachpixel position of the image data is, (2) determine a lesion area and atype of lesion by determining a type of lesion of each pixel in units ofpixels of the image data. Alternatively, correct answer data for oneimage may be specified as a disease name or an image diagnostic name,and (3) the disease name may be specified from the image data.

Next, a second embodiment will be described. The second embodiment isdifferent from the first embodiment in the method of determining thecorrect answer data. Since a schematic configuration of the learningassistance system 1 is the same as that of the first embodiment,detailed description thereof will be omitted. FIG. 7 is a block diagramillustrating a schematic configuration of a terminal device 10 and alearning assistance device 20 according to the second embodiment. Thesame configurations as those of the first embodiment are denoted by thesame reference numerals as those of the first embodiment, detaileddescription thereof will be omitted, and only different configurationswill be described.

As illustrated in FIG. 7, the terminal device 10 includes adiscriminator acquisition unit 12, a discrimination result acquisitionunit 13, a learning unit 14, and a learned discriminator output unit 15.The learning assistance device 20 includes a learned discriminatoracquisition unit 22, a discriminator storage unit 23, a correct answerdata acquisition unit 24 a, a correct answer data storage unit 25, alearning unit 26, and a discriminator output unit 27. A configuration ofthe terminal device 10 is the same as that of the first embodimentexcept that the correct answer data acquisition unit 24 a of thelearning assistance device 20 includes an evaluation unit 28 and anevaluation image storage unit 29.

In the case where the correct answer data is determined by majorityvote, sufficient learning may be performed in each terminal device 10 asin the first embodiment, but for example, in a case where adiscriminator not additionally learned in the terminal device 10 isreceived as the learned discriminator or in a case where a learneddiscriminator with a small number of additional learnings is used, adetermination result is likely to be the same for the same input imagedata P, and a determination result of the discriminator in which suchlearning is not sufficiently performed is highly likely to be thecorrect answer data.

Therefore, the learning assistance device 20 evaluates the learneddiscriminators collected from the respective terminal device 10 inadvance. The learning assistance device 20 sets a plurality of cases ofdisease covering a representative case pattern and also having correctanswer data for image data as an evaluation image set SET, and theevaluation unit 28 evaluates the learned discriminators sent from therespective terminal devices 10 using the image set SET, and determinesthe weight of the discriminator according to a height of the correctanswer rate. Further, since software intended for medical purposes is atarget of the Pharmaceutical and Medical Device Act (the revisedPharmaceutical Affairs Act), the software is required to meet acriterion prescribed in the Pharmaceutical and Medical Device Act.Therefore, it is preferable for an evaluation image set SET formedthrough a combination of a plurality of images in which the criterionprescribed in the Pharmaceutical and Medical Device Act can be evaluatedto be stored in the storage (the evaluation image storage unit 29) inadvance. However, this evaluation image set SET is not sufficient foruse in deep learning.

The weights are determined for the respective learned discriminatorscollected from the terminal device 10 according to the correct answerrate of the evaluation image set, the weights of the learneddiscriminators having the same result among the discrimination resultsare added, and the discrimination result having the largest added weightis set as correct answer data of the input image.

Since a flow of the deep learning process is the same as that of thefirst embodiment, the flow will be omitted.

Next, a third embodiment will be described. The third embodiment isdifferent from the first and second embodiments in the method ofdetermining the correct answer data. Since a schematic configuration ofthe learning assistance system 1 is the same as that of the firstembodiment, detailed description thereof will be omitted. FIG. 8 is ablock diagram illustrating a schematic configuration of a terminaldevice 10 and a learning assistance device 20 according to the thirdembodiment. The same configurations as those of the first embodiment aredenoted by the same reference numerals as those of the first embodiment,detailed description thereof will be omitted, and only differentconfigurations will be described.

As illustrated in FIG. 8, the terminal device 10 includes adiscriminator acquisition unit 12, a discrimination result acquisitionunit 13, a learning unit 14, and a learned discriminator output unit 15a. The learning assistance device 20 includes a learned discriminatoracquisition unit 22 a, a discriminator storage unit 23, a correct answerdata acquisition unit 24 b, a correct answer data storage unit 25, alearning unit 26, and a discriminator output unit 27. The learneddiscriminator output unit 15 a of the terminal device 10, and thelearned discriminator acquisition unit 22 a and the correct answer dataacquisition unit 24 b of the learning assistance device 20 are differentfrom those of the first embodiment.

As in the second embodiment, even in a case where the performance of thelearned discriminator is evaluated, evaluation results using theevaluation image set may be insufficient in a case where there is aproblem with the number of cases of disease for evaluation of thelearning assistance device 20 or coverage of the cases of disease.However, in a case where the learned discrimination learns a certainnumber of pieces of image correct answer data, it can be presumed thatthe performance is improving as appropriate.

Therefore, in a case where the learned discriminator output unit 15 a ofthe terminal device 10 transmits the parameter of the learneddiscriminator, the learned discriminator output unit 15 a of theterminal device 10 transmits the number of pieces of image correctanswer data learned by the learned discriminator to the learningassistance device 20.

The learned discriminator acquisition unit 22 a of the learningassistance device 20 receives the number of pieces of image correctanswer data learned by the learned discriminator at each terminal device10 together in a case where the learned discriminator acquisition unit22 a of the learning assistance device 20 receives the parameter. Theweight is determined so that the weight increases as the number ofpieces of image correct answer data learned by the correct answer dataacquisition unit 24 b and the learned discriminators of each terminaldevice 10 increases. The weights of the learned discriminators havingthe same discrimination result are added, and the discrimination resultwith the largest added weight is set as the correct answer data of theinput image.

Since a flow of deep learning process is the same as that of the firstembodiment, description thereof is omitted.

Next, a fourth embodiment will be described. The fourth embodiment isdifferent from the first, second, and third embodiments in the method ofdetermining the correct answer data. Since a schematic configuration ofthe learning assistance system 1 is the same as that of the firstembodiment, detailed description thereof will be omitted. FIG. 9 is ablock diagram illustrating a schematic configuration of a terminaldevice 10 and a learning assistance device 20 according to the fourthembodiment. The same configurations as those of the first embodiment aredenoted by the same reference numerals as those of the first embodiment,detailed description thereof will be omitted, and only differentconfigurations will be described.

As illustrated in FIG. 9, the terminal device 10 includes adiscriminator acquisition unit 12, a discrimination result acquisitionunit 13, a learning unit 14, and a learned discriminator output unit 15b. The learning assistance device 20 includes a learned discriminatoracquisition unit 22 b, a discriminator storage unit 23, a correct answerdata acquisition unit 24 c, a correct answer data storage unit 25, alearning unit 26, and a discriminator output unit 27. The learneddiscriminator output unit 15 b of the terminal device 10 and the learneddiscriminator acquisition unit 22 b and the correct answer dataacquisition unit 24 c of the learning assistance device 20 are differentfrom those of the first embodiment.

The number of pieces of image correct answer data for each type of casesof disease is biased due to the characteristics of the medical facility,the regional nature, or the like, and the number of pieces of imagecorrect answer data is small in any kind of diseases even though thenumber of all pieces of image correct answer data is large, theperformance is likely not to be improved in the disease. Therefore, thenumber of pieces of image correct answer data learned by the learneddiscriminator of each medical facility is received from each terminaldevice 10 for each type of cases of disease.

Therefore, in a case where the learned discriminator output unit 15 b ofthe terminal device 10 transmits the parameter of the learneddiscriminator, the learned discriminator output unit 15 b of theterminal device 10 transmits the number of pieces of image correctanswer data for each type of cases of disease learned by the learneddiscriminator to the learning assistance device 20. Specifically, it isdetermined which case of disease the image correct answer data learnedby the learning unit 14 of the terminal device 10 relates to, forexample, on the basis of a DICOM tag attached to the image of imagecorrect answer data, and the number of learned image correct answer datais changed for each type of cases of disease. The type of cases ofdisease is classified by disease name (which may be an image diagnosticname in case of image inspection) or a type of disease name. In a casewhere a plurality of organs are collectively processed by onediscriminator, an organ name may be used.

The learned discriminator acquisition unit 22 b of the learningassistance device 20 receives the number of pieces of image correctanswer data for each type of cases of disease learned by the learneddiscriminator at each terminal device 10 in a case where the learneddiscriminator acquisition unit 22 b of the learning assistance device 20receives the parameter.

In the correct answer data acquisition unit 24 c, it is estimated thatthe performance of the learned discriminator is higher as the number oflearned image correct answer data is larger. To reflect this, the numberof pieces of image correct answer data learned by each facility for eachtype of cases of disease is counted for the learned discriminator ofeach terminal device 10, and, the weight for each type of cases ofdisease is determined for each learned discriminator so that weight isincreased as the number increases. In addition, weights of the learneddiscriminators having the same discrimination result are added incorrespondence to the type of cases of disease of the input image, andthe discrimination result having the largest added weight is set as thecorrect answer data of the input image.

Since a flow of deep learning process is the same as that of the firstembodiment, description thereof will be omitted.

With a scheme according to the embodiment, it is possible to evaluatethe learned discriminator in consideration of, the number of pieces ofimage correct answer data of each medical facility, the type of cases ofdisease, and the like.

Further, the evaluation image set according to the second embodiment maybe set as an evaluation image set capable of evaluating a discriminatorfor each type of disease, and the weight of the learned discriminator ofeach terminal device may be determined according to the correct answerrate for each type of disease. The weights of the learned discriminatorshaving the same discrimination result may be added according to the typeof cases of disease of the input image, and the discrimination resulthaving the largest added weight may be set as the correct answer data ofthe input image.

Further, although the weight is automatically determined in the fourthembodiment, the weight may be determined manually and stored in thelearning assistance device 20 in advance in consideration of importanceof the facility, a confident disease of each facility, or the like.

Although the embodiment in which the learning assistance device 20 andthe terminal device 10 are connected via the network has been describedin the above description, an image processing program incorporating adiscriminator functioning as an actually operated discriminator and alearning program incorporating a discriminator functioning as a learningdiscriminator may be stored in a recording medium such as a DVD-ROM anddistributed to each medical institution, instead of over the network.

In this embodiment, the discriminator acquisition unit 12 of theterminal device 10 reads the image processing program and the learningprogram from the DVD-ROM to the terminal device 10 and installs theimage processing program and the learning program, and reads theidentification information ID of the image correct answer data used forlearning of the learning discriminator from the recording medium.Further, the learned discriminator output unit 15 of the terminal device10 records the parameter of the learned discriminator in the DVD-ROM,and distributes the parameter to an operator of the learning assistancedevice 20 by mailing or the like.

Further, the learned discriminator acquisition unit 22 of the learningassistance device 20 reads the parameters of the learned discriminatorrecorded on the DVD-ROM. Furthermore, the discriminator output unit 27of the learning assistance device 20 records the image processingprogram and the learning program on a DVD-ROM and sends the imageprocessing program and the learning program to an operator of theterminal device 10 by mailing or the like.

As described in detail above, in the present invention, accurate correctanswer data of an image is automatically generated using a discriminatorof which the performance has been improved using the medical imagesstored in each medical institution. Thus, it is possible to use a largeamount of medical images for deep learning.

Although the case where the learning assistance device and the terminaldevice function on a general-purpose computer has been described above,a dedicated circuit such as an application specific integrated circuit(ASIC) or field programmable gate arrays (FPGA) that permanently storesa program for executing some of functions may be provided.Alternatively, a program instruction stored in a dedicated circuit and aprogram instruction executed by a general-purpose CPU programmed to usea program of a dedicated circuit may be combined. As described above,the program instructions may be executed through any combination ofhardware configurations of the computer.

What is claimed is:
 1. A learning assistance system comprising alearning assistance device and terminal devices connected over anetwork, wherein each of the terminal devices comprises a processorconfigured to output a learned discriminator to the learning assistancedevice over the network, the learned discriminator being obtained bycausing a learning discriminator to perform learning using an image andcorrect answer data thereof that are stored in a database of each of theterminal devices, the learning assistance device comprises a processorconfigured to: acquire learned discriminators from the terminal devicesover the network; acquire a discrimination results obtained by causingthe learned discriminators to discriminate the same input image, each ofthe discrimination results being obtained from each of the learneddiscriminators; determine correct answer data of the same input image onthe basis of the discrimination results; and output a new learningdiscriminator to each of the terminal devices over the network, the newlearning discriminator being obtained by causing a learningdiscriminator to perform learning using the same input image and thedetermined correct answer data.
 2. The learning assistance systemaccording to claim 1, wherein the processor of each of the terminaldevices device is configured to obtain a new learned discriminator bycausing the new learning discriminator to perform learning using animage and correct answer data thereof that are stored in the database ofeach of the terminal devices.
 3. The learning assistance systemaccording to claim 1, wherein the processor of each of the terminaldevices is configured to: acquire a learned actually operateddiscriminator learned using the same image and correct answer data ofthe same image as those of the learning discriminator; acquire adiscrimination result of discriminating an image that is adiscrimination target using the actually operated discriminator.
 4. Thelearning assistance system according to claim 1, wherein the processorof each of the terminal devices is further configured to acquire anactually operated discriminator from the learning assistance device overa network, the actually operated discriminator learning the same imageand correct answer data of the same image as those of the learningdiscriminator, and acquire a discrimination result of discriminating animage that is a discrimination target using the actually operateddiscriminator.
 5. The learning assistance system according to claim 1,wherein the processor of each of the terminal devices is furtherconfigured to: acquire a new actually operated discriminator from thelearning assistance device over a network, the actually operateddiscriminator learning the same input image and the determined correctanswer data, and acquire a discrimination result of discriminating animage that is a discrimination target using the new actually operateddiscriminator.
 6. The learning assistance system according to claim 1,wherein the processor of the learning assistance device is configured tooutput an actually operated discriminator learning the same input imageand the determined correct answer data.
 7. The learning assistancesystem according to claim 1, wherein the processor of the learningassistance device is configured to determine a discrimination resulthaving the largest number of same results among the discriminationresults, as correct answer data of the same input image.
 8. The learningassistance system according to claim 1, wherein the processor of thelearning assistance device is configured to: determine a weight of eachof the learned discriminators according to each of the terminal devices;add weights of learned discriminators having the same result among thediscrimination results; and set a discrimination result having thelargest added weight as correct answer data of the same input image. 9.The learning assistance system according to claim 1, wherein theprocessor of the learning assistance device is configured to: determinea weight of each of the learned discriminators according to the numberof pieces of correct answer data learned by the learned discriminator ateach of the terminal devices; add weights of learned discriminatorshaving the same result among the discrimination results; and set adiscrimination result having the largest added weight as correct answerdata of the same input image.
 10. The learning assistance systemaccording to claim 1, wherein the processor of the learning assistancedevice is configured to: determine weights for types of cases of diseaseof the image learned by each of the learned discriminators with respectto each of the learned discriminators, add weights corresponding to thetypes of cases of disease of the image of learned discriminators havingthe same result among the discrimination results; and set adiscrimination result having the largest added weight as correct answerdata of the same input image.
 11. The learning assistance systemaccording to claim 1, wherein the processor of the learning assistancedevice is configured to: evaluate a correct answer rate using an imageset including images with respect to each of the learned discriminators;determine a weight of each of the learned discriminators according tothe correct answer rate; add weights of learned discriminators havingthe same discrimination results among the discrimination results; andset a discrimination result having the largest added weight as correctanswer data of the same input image.
 12. A terminal device comprising aprocessor configured to: acquire a learning discriminator, and a learnedactually operated discriminator learned using the same image and correctanswer data of the same image as those of the learning discriminator;acquire a discrimination result of discriminating an image that is adiscrimination target using the actually operated discriminator; andoutput a learned discriminator to a learning assistance device, thelearned discriminator being obtained by causing the learningdiscriminator to perform learning using an image and correct answer datathereof that are stored in a database of the terminal device.
 13. Theterminal device according to claim 12, wherein the processor isconfigured to acquire the learning discriminator and the actuallyoperated discriminator from a learning assistance device over a network,and the processor is configured to send and output the learneddiscriminator over the network.
 14. The terminal device according toclaim 12, wherein the processor is configured to receive a new learningdiscriminator, the new learning discriminator being obtained by causinga the learning discriminator to perform learning using the same inputimage and determined correct answer data, the determined correct answerdata being determined by the learning assistance device on the basis ofthe discrimination results, and the discrimination results beingacquiring by causing learned discriminators including the learneddiscriminator to discriminate the same input image.
 15. The terminaldevice according to claim 14, wherein the processor is configured toreceive a new actually operated discriminator from the learningassistance device, the actually operated discriminator learning the sameinput image and the determined correct answer data.
 16. A method foroperating a learning assistance system comprising a learning assistancedevice and terminal devices connected over a network, the methodcomprising: acquiring learned discriminators from the terminal devicesover the network, each of the learned discriminators being obtained bycausing a learning discriminator to perform learning using an image andcorrect answer data thereof; acquiring a discrimination results obtainedby causing the learned discriminators to discriminate the same inputimage, each of the discrimination results being obtained from each ofthe learned discriminators; determining correct answer data of the sameinput image on the basis of the discrimination results; and outputting anew learning discriminator to each of the terminal devices over thenetwork, the new learning discriminator being obtained by causing alearning discriminator to perform learning using the same input imageand the determined correct answer data.
 17. A non-transitorycomputer-readable recording medium storing therein a learning assistanceprogram causing a computer to perform the method according to claim 16.18. A method, comprising: acquiring a learning discriminator, and alearned actually operated discriminator learned using the same image andcorrect answer data of the same image as those of the learningdiscriminator; acquiring a discrimination result of discriminating animage that is a discrimination target using the actually operateddiscriminator; and outputting a learned discriminator to a learningassistance device, the learned discriminator being obtained by causingthe learning discriminator to perform learning using an image andcorrect answer data thereof that are stored in a database of theterminal device.
 19. A non-transitory computer-readable recording mediumstoring therein a program causing a computer to perform the methodaccording to claim 18.